A Review on Deep Learning in UAV Remote Sensing

Deep Neural Networks (DNNs) learn representation from data with an impressive capability, and brought important breakthroughs for processing images, time-series, natural language, audio, video, and many others. In the remote sensing field, surveys and literature revisions specifically involving DNNs algorithms' applications have been conducted in an attempt to summarize the amount of information produced in its subfields. Recently, Unmanned Aerial Vehicles (UAV) based applications have dominated aerial sensing research. However, a literature revision that combines both "deep learning" and "UAV remote sensing" thematics has not yet been conducted. The motivation for our work was to present a comprehensive review of the fundamentals of Deep Learning (DL) applied in UAV-based imagery. We focused mainly on describing classification and regression techniques used in recent applications with UAV-acquired data. For that, a total of 232 papers published in international scientific journal databases was examined. We gathered the published material and evaluated their characteristics regarding application, sensor, and technique used. We relate how DL presents promising results and has the potential for processing tasks associated with UAV-based image data. Lastly, we project future perspectives, commentating on prominent DL paths to be explored in the UAV remote sensing field. Our revision consists of a friendly-approach to introduce, commentate, and summarize the state-of-the-art in UAV-based image applications with DNNs algorithms in diverse subfields of remote sensing, grouping it in the environmental, urban, and agricultural contexts.Comment: 38 pages, 10 figure

Estimation of Phytoplankton Chlorophyll-a Concentrations in the Western Basin of Lake Erie Using Sentinel-2 and Sentinel-3 Data

Algae blooms have been a serious problem in coastal and inland water bodies across Canada. The temporal and spatial variability of algae blooms makes it difficult to use in situ monitoring of the lakes. This study aimed to evaluate the potential of Sentinel-3 Ocean and Land Color Instrument (OLCI) and Sentinel-2 Multispectral Instrument (MSI) data for monitoring algal blooms in Lake Erie. Chlorophyll-a (Chl-a)-related products of these sensors were tested by using the Great Lakes Chl-a NOAA’s monitoring data over summer 2016 and 2017, respectively. Our results show that while fluorescent light height (FLH) algorithm and models are limited to lakes with Chl-a  20 mg/m3 and Sentinel-2 MCI for Chla > 8 mg/m3. Top of atmosphere (TOA) radiances showed a significantly better correlation with in situ data compared to TOA reflectance, which may be related to the poor pixel identification during the process of pixel flagging affected by the complexity of Case-2 water. Sentinel-2 MCI achieves better performance for Chl-a retrieval (R2 = 0.92) than the existing methods. However, the FLH algorithms outperformed negative reflectance due to the shift of reflectance peak to longer wavelengths along with increasing Chl-a values

A Comparative Study of Deep Learning Approaches to Rooftop Detection in Aerial Images

This paper investigates the deep neural networks for rapid and accurate detection of building rooftops in aerial orthoimages. The networks were trained using the manually labeled rooftop vector data digitized on aerial orthoimagery covering the Kitchener-Waterloo area. The performance of the three deep learning methods, U-Net, Fully Convolutional Network (FCN), and Deeplabv3+ were compared by training, validation, and testing sets in the dataset. Our results demonstrated that DeepLabv3+ achieved 63.8% in Intersection over Union (IoU), 77.8% in mean IoU (mIoU), 74% in precision, and 78% in F1-score. After improving the performance with focal loss, training loss was greatly cut down and the convergence rate experienced a significant growth. Meanwhile, rooftop detection also achieved higher performance, as Deeplabv3+ reached 93.6% in average pixel accuracy, with 65.4% in IoU, 79.0% in mIoU, 77.6% in precision, and 79.1% in F1-score. Lastly, in order to evaluate the effects of data volume, by changing data volume from 100% to 75% and 50% in ablation study, it shows that when data volume decreased, the performance of extraction also got worse, with IoU, mIoU, precision, and F1-score also mostly decreased

High-Resolution Terrain Modeling Using Airborne LiDAR Data with Transfer Learning

This study presents a novel workflow for automated Digital Terrain Model (DTM) extraction from Airborne LiDAR point clouds based on a convolutional neural network (CNN), considering a transfer learning approach. The workflow consists of three parts: feature image generation, transfer learning using ResNet, and interpolation. First, each point is transformed into a featured image based on its elevation differences with neighboring points. Then, the feature images are classified into ground and non-ground using ImageNet pretrained ResNet models. The ground points are extracted by remapping each feature image to its corresponding points. Last, the extracted ground points are interpolated to generate a continuous elevation surface. We compared the proposed workflow with two traditional filters, namely the Progressive Morphological Filter (PMF) and the Progressive Triangulated Irregular Network Densification (PTD). Our results show that the proposed workflow establishes an advantageous DTM extraction accuracy with yields of only 0.52%, 4.84%, and 2.43% for Type I, Type II, and the total error, respectively. In comparison, Type I, Type II, and the total error for PMF are 7.82%, 11.60%, and 9.48% and for PTD 1.55%, 5.37%, and 3.22%, respectively. The root means square error (RMSE) for the 1 m resolution interpolated DTM is only 7.3 cm. Moreover, we conducted a qualitative analysis to investigate the reliability and limitations of the proposed workflow

Waterloo Building Dataset: A city-scale vector building dataset for mapping building footprints using aerial orthoimagery

Automated building footprint extraction is an important area of research in remote sensing with numerous civil and environmental applications. In recent years, deep learning methods, when trained on appropriate datasets, have far surpassed classical algorithms. In this paper, we present the Waterloo Building Dataset for building footprint extraction from very-high-spatial-resolution aerial orthoimagery. Our dataset covers the Kitchener-Waterloo area in Ontario, Canada, contains 117,000 manually labelled buildings, and extends over an area of 205.8 km2. At a spatial resolution of 12 cm, it is the highest resolution publicly available building footprint extraction dataset in North America. We provide extensive benchmarks of commonly used deep learning architectures trained on our dataset which can be used as baseline for future models. We also identify a key challenge in aerial orthoimagery building footprint extraction which we hope can be addressed in future research.The accepted manuscript in pdf format is listed with the files at the bottom of this page. The presentation of the authors' names and (or) special characters in the title of the manuscript may differ slightly between what is listed on this page and what is listed in the pdf file of the accepted manuscript; that in the pdf file of the accepted manuscript is what was submitted by the author

A region-based deep learning approach to instant segmentation of aerial orthoimagery for building rooftop detection

Updated building information plays an important role in many fields such as environmental monitoring, disaster assessment and the creation of base maps for urban planning. High-resolution images captured from earth observing satellites and airborne platforms provide valuable data covering large areas at high temporal frequencies. In recent years, deep neural networks have shown great potential in semantic segmentation of earth observation images for building detection. In this paper, we approach building rooftop detection as an instance segmentation problem and propose a region-based deep learning approach to building rooftop detection based on the Mask R-CNN framework. Our study indicates that searching for suitable hyper-parameters result in considerable improvements in deep learning models. We find that hyper-parameter optimization could be mandatory in some cases, since in our experiments, the baseline Mask R-CNN achieved an unacceptable performance when compared to other methods. Our optimized Mask R-CNN, on the other hand, achieves a precision, recall, and F1-score of 92%, 86.6%, and 89.1%, respectively. Furthermore, we show that by using a region-based instance segmentation model, we avoid the speckle-like errors sometimes found in semantic segmentation models. These results demonstrate the effectiveness of our region-based deep learning approach to building rooftop detection.The accepted manuscript in pdf format is listed with the files at the bottom of this page. The presentation of the authors' names and (or) special characters in the title of the manuscript may differ slightly between what is listed on this page and what is listed in the pdf file of the accepted manuscript; that in the pdf file of the accepted manuscript is what was submitted by the author